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■ Abstract 



Properties of scalar quantization with rth power distortion and constrained Renyi entropy of 
ff^ ' order a G (0, 1) are investigated. For an asymptotically (high-rate) optimal sequence of quantizers, 

the contribution to the Renyi entropy due to source values in a fixed interval is identified in terms of 
the "entropy density" of the quantizer sequence. This extends results related to the well-known point 
density concept in optimal fixed-rate quantization. A dual of the entropy density result quantifies the 
tyj \ distortion contribution of a given interval to the overall distortion. The distortion loss resulting from 

a mismatch of source densities in the design of an asymptotically optimal sequence of quantizers is 
also determined. This extends Bucklew's fixed-rate {a ~ 0) and Gray et al.'s variable-rate (a — 1) 
mismatch results to general values of the entropy order parameter a. 



Index Terms: Asymptotic quantization theory, distortion density, entropy density, quantizer mismatch, 
Renyi-entropy. 

1 Introduction 

Asymptotic quantization theory studies the performance of quantizers of a fixed dimension in the limit of 
high rates (low distortion). This approach complements Shannon's rate-distortion theory where optimal 
codes of a fixed rate (distortion) are investigated as the dimension becomes asymptotically large. Panter 
and Dite ||26]| were the first to derive a formula for the mean square distortion of optimum scalar quan- 
tizers as the number of quantization levels becomes asymptotically large. Zador's classic work |29] for 
vector quantizers determined the asymptotic behavior of the minimum quantizer distortion under a con- 
straint on either the log-cardinality of the quantizer codebook (fixed-rate quantization) or the Shannon 
entropy of the quantizer output (entropy-constrained quantization). Zador's results were later clarified 
and generalized by Bucklew and Wise [6] and Graf and Luschgy II121I for the fixed-rate case, and by 
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Gray et al. f\M for the entropy-constrained case. Gray and Neuhoff lfT6l provide a historical overview 
of related results. 

One way to unify and extend the fixed and variable-rate results is to define the quantizer's rate by the 
Renyi entropy of order a of its output. This generalized rate concept includes the traditional rate defi- 
nitions as special cases since a = corresponds to fixed-rate quantization, while setting a = 1 yields 
variable-rate quantization. This approach was first suggested in [13] as an alternative to the Lagrangian 
rate definition considered there which simultaneously controls codebook size and output (Shannon) en- 
tropy. Further motivation for using Renyi entropy as quantization rate can be obtained from axiomatic 
considerations ll27l[T]|, as well as from the operational role of the Renyi entropy in variable-length lossless 
coding il [HI 12. 

The theory of quantization with Renyi o-entropy constraint has recently been explored in |[T8l[T9ll20l 
I2TI . In particular, |[T9] derived the sharp asymptotic behavior of the rth power distortion of optimal d- 
dimensional vector quantizers for a > 1 + r/d. In 1(211 the technically more challenging a < 1 case was 
considered and the asymptotically optimal rth power distortion was determined for scalar quantization 
{d = 1) and a fairly large class of source densities. Thus, at least for scalar quantization, only the case 
a G (1, 1 + r) remains open, and it is conjectured in flT] that the main result there remains valid in this 
range of the parameter a. 

In addition to the asymptotic behavior of the optimal quantizer performance, asymptotic quantiza- 
tion theory has also been concerned with more subtle properties of (asymptotically) optimal quantizers. 
One such property is the existence, for a sequence of quantizers, of the so-called quantizer point den- 
sity function, loosely defined as a probability density which, when integrated over a region, gives the 
fraction of the quantization levels contained in that region. More formally, a point density, if exists, is 
the probability density function of the limit distribution of the output levels of a sequence of quantiz- 
ers. Point densities and the closely related companding quantizers have been instrumental in the early 
pioneering investigations into optimal scalar and vector quantization ||3] |26j |22j [TOl (see also ["241 for a 
rigorous reformulation of Bennett's result for the vector case and |fT6l| for the history of these results). 
Bucklew [7 | was the first to rigorously establish the existence of the point density function for an asymp- 
totically optimal sequence of fixed-rate quantizers. To our knowledge, no such rigorous result is known 
for variable-rate quantization. The concept of quantizer point density has been very useful in analyzing 
the performance of quantizers in a distributed setting (e.g. ||28ll25l ). 

Asymptotic quantization theory has also been successful in providing mismatch results that quantify 
the loss in performance when a sequence of quantizers that is asymptotically optimal for one source 
is applied to a different source. Mismatch results are theoretically important and in practice they may 
provide a means for quantifying the performance of code designs that are based on source models esti- 
mated from data. For fixed-rate vector quantization Bucklew ||71 was the first to prove such a rigorous 
mismatch result. The variable-rate analog of this result was proved in fTSl where connections with mis- 
match results in rate-distortion theory and robust lossy coding were also pointed out. More recently, Na 
||23l determined sharp asymptotic formulas for variance-mismatched scalar quantization of Laplacian 
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sources. 

In this paper we extend some of the more refined results of fixed and variable-rate asymptotic quan- 
tization theory in the framework of quantization with Renyi entropy constraint of order a E (0, 1). The 
concept of a quantizer point density is a problematic one for (Renyi) entropy-constrained quantization 
since (near) optimal quantizers can have an arbitrarily large number of levels in any bounded region. 
Instead, we investigate the Renyi entropy contribution of a given interval to the overall rate. One of our 
main results, Theorem [2l shows that for a large class of source densities and an asymptotically optimal 
sequence of quantizers, this contribution can be quantified by the so called entropy density of the se- 
quence. A dual of this result. Corollary [TJ quantifies the distortion contribution of a given region to the 
overall distortion in terms of the so-called distortion density. Interestingly, it turns out that the entropy 
and distortion densities are equal in the cases we investigate (Remark |5]). Our other main contribution. 
Theorem |3l is a mismatch formula for a sequence of asymptotically optimal Renyi entropy constrained 
scalar quantizers. From our density and mismatch results we can recover the known results for the 
traditional rate definitions by formally setting a = or a = 1. 

The rest of the paper is organized as follows. In the next section we formulate the quantization 
problem and give a somewhat informal overview of our results in the context of prior work. In Section [3] 
the entropy and distortion density results are presented and proved. The mismatch problem is considered 
in Section m Concluding remarks are given in Section |51 



2 Preliminaries and overview of results 



2.1 Renyi entropy and quantization 



We begin with the definition of Renyi entropy of order a. Let N := {1, 2, . . .} and letp = {pi,P2, ■ ■ ■) € 
[0, 1]^ be a probability vector, i.e. Yli^iPi — 1- ^ (0) \ {!}> "^^e Renyi entropy of order 

a, ^{p) G [0, cx)], is defined as (see ETIl or ifB) 



Remark 1. All logarithms in this paper have base e. Setting 0° := 0, we can extend the definition to 
a = 0, obtaining 

H^{p) = log (card{i G N : > 0}) (1) 

where card denotes cardinality. Also, using the convention OlogO := 0, it is easy to see that letting 
a — )• 1 yields the regular (Shannon) entropy of p: 



H^{p) := lim H'^ip) = - "S^ pi log pi 



i=l 
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assuming H°'{p) is finite for some a < 1. 

Let X be a real-valued random variable with distribution ^. Let I C N be an index set (thus I is 
either finite or countably infinite) and 5 = {Sj : i G 1} a Borel measurable partition of the real line M. 
Moreover let C = {q : i e 1} be set of distinct points in M. Then {Si, Ci)igi defines a (scalar) quantizer 
5 : M ^ C such that 

q{x) = Ci if and only if x G Si. 

We call C the codebook and the q the codepoints (or quantization levels). Each Sj € <S is called codecell. 
Clearly C = (/(M) is the range of q and 

S = {q-\z) : z q{M.)} 

where q~'^{z) = {x G M : q{x) = z}. Let Q denote the set of scalar quantizers, i.e., the set of all 
Borel-measurable mappings g : M — > M with a countable range. The discrete random variable q{X) is a 
quantized version of the random variable X. With any enumeration {ii, ^2, . . .} of I we define 

H^{q)=H"{f,{Si,),f,{Si,),...) 

as the Renyi entropy of order a of g with respect to ji. Thus (g) is the log-cardinality of the codebook 
of q (we assume without loss of generality that each codecell of q has positive probability) and H^{q) is 
the Shannon entropy of the quantizer output. 

For r > 1 and g G Q we measure the approximation error between X and q{X) by the the rth power 
distortion defined by 

D,{q)=E\X-q{X)\' = J \x - q{x)\' d^^{x). 

For any i? > we define 

= mi{D^{q) : q G Q,H^{q) < R} 

the optimal quantization distortion of n under Renyi entropy constraint R. We call a quantizer q optimal 
for /i under the entropy constraint R if D^{q) = D'^{R) and H^{q) < R. In particular, D^{R) is the 
minimum distortion of any quantizer with codebook size not exceeding e^, while Dj^{R) is the minimum 
distortion under Shannon entropy constraint R. 

In the rest of this paper all distributions to be quantized will be absolutely continuous with respect 
to the Lebesgue measure A on the real line. If such a distribution /u has probability density function g, 
then we will use the notation /i = gX. We denote by supp(//) the support of fi (the smallest closed set 
whose complement has zero /i measure). If /u = gX, then we define supp((7) = supp(//). We will also 
assume throughout the paper that the rth moment of fi is finite, i.e. / d^{x) < oo. This condition is 
sufficient (but not necessary) for D'^{R) to be finite for all R > 0. 

It has been shown in ||20l that under the above conditions, the set of all quantizers Q in the definition 
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of D'^ can be replaced by the set of quantizers having finitely many codecells, each of which is an 
interval. In view of this, we will assume throughout the whole paper that the codecells of every quantizer 
g G Q are intervals (but we do not restrict the number of codecells to be finite) and each codepoint is 
contained in the interior of the associated codecell. 



2.2 Asymptotic optimality and conditional distributions 

The main result of ll2TI implies that under suitable assumptions on the source density g, for all a < 1, 



\\m e^^D^JR) = —-^-—e''^^'^3^ (2) 
R^oo ' (1 + r)2^ 



where Bi = i,zH±2l and 



is the Renyi differential entropy of order /3i of g. 

We can formally recover Zador's classical results ||29l in the scalar setting from Q. Letting a = 0, 
we have /3i = -rr-, and e^'^'^^^^^ = (/ g~ dX)^'^^ = H^H^, yielding Zador's formula for fixed-rate 

' 1+?' 

scalar quantization. For a = 1, we have /3i = 1 and e''^'^^(9) = Q^Ka) ^ where h{g) = — J g log g dX is 
the Shannon differential entropy of g, and ^ becomes Zador's formula for variable-rate scalar quanti- 
zation. In view of Q we call a sequence of quantizers (gn)ngN asymptotically optimal if H^{qn) oo 
and 

n^oo (^1 + r)2^ 

Suppose / is a bounded interval with positive /x probability. We denote by /u(-|/) the conditional 
distribution for fj, given / and by gj the corresponding conditional density (so that fJ,{-\I) = giX). We 
show in Theorem |2] that for a S (0, 1) any quantizer sequence {qn) that is asymptotically optimal for ^ 
is also asymptotically optimal for fJ,{-\I), i.e., 

lim e'-^M I^)(^")z),(.|,)(,„) = j:r^e^'''^3'\ (3) 

n^oo f-v I / (f + rlz 

Although this result is not very surprising, it will be very useful in establishing further, more subtle 
properties of asymptotically optimal quantizers. 

2.3 Entropy and distortion densities 

Let Nn{I) denote the number of codepoints of contained in an interval /. Let a = and let (qn) be 
a sequence of asymptotically optimal n-level quantizers (so that H^{qn) = logn). Specialized to the 
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scalar case, one important result of Bucklew Q shows that 

lim Ml = IU!±J!1. (4) 

1 1 

Thus the probability density 51+' / J (71+'^ dA can be interpreted as the point density function for the 
codepoints of asymptotically optimal quantizers (see also |[T2l Thm. 7.5]). Point densities are useful in 
gaining insight into the structure of (asymptotically) optimal quantizers and can be used to construct such 
quantizers via a companding construction. 

Unfortunately, no rigorous point density results are known for a = 1. In fact, even the definition 
of a point density function is problematic for entropy-constrained quantization since for sources with a 
density, at any rate i? > there exist near-optimal quantizers that have an arbitrarily large number of 
codepoints contained in a given bounded interval. Thus an analog of ^ cannot hold for an arbitrary 
sequence of asymptotically optimal quantizers, although heuristic arguments indicate that under some 
structural restrictions asymptotically optimal variable-rate quantizers have a uniform point density (see, 

e.g., mm). 

To define a tractable analog of the point density function, recall that denotes the conditional 

distribution for fi given /. In view of ([T]), we have Nn{I) = e^''' l^)^''"^ and n = e^°^''"\ Thus the 
fraction of codepoints contained in / on the left hand side of (01) can be rewritten as 

This ratio represents the relative contribution of the interval / to the total Renyi entropy of order a = 0. 

The interpretation in ^ motivates us to define the Renyi entropy contribution of an interval / in 
a similar way for general a. In Theorem [2j we identify the limit of this entropy contribution: Under 
appropriate conditions on the source density, for any a G (0, 1) and asymptotically optimal sequence 
(qn), we have 

It is easy to see that ^ reduces to the traditional point density result Q for q = 0. 

In Corollary [T] we present an almost immediate consequence of (O and Q which concerns the dis- 
tortion contribution of an arbitrary finite interval I: 

j.^ Jj \x - qn{x)\'' fj,{dx) _ fj g^^ dX 

Thus the probability density g^'^ j J g^^ dX can be interpreted as either the (Renyi) entropy density or the 
distortion density of any asymptotically optimal quantizer sequence [qn)- 
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2.4 Mismatch 



For scalar quantization Bucklew's fixed-rate mismatcli result [7, Tlim. 2] can be stated as follows: If a 
sequence of n-level quantizers that is asymptotically optimal for a source with distribution ji = gX 
is applied to a source with distribution u = fX, then (under some assumptions on g and /) 



1 f f 
lim vJ'D^{qn) = j— — r— / — dX 
n^oo (1 + r)2' J gl 



1 



where g^ = g~ / J g~ dX is the optimal point density function for fi = gX from This is a 
generalization of a classical result of Bennett IS who considered companding quantization and mean 
square distortion. The integral on the right hand side is often called Bennett's integral. In view of (HJl, 
and after some calculations, we obtain that the asymptotic performance loss due to mismatch is 



lim = grVi+riMlg*) (7) 

n->-oo Z)0(log n) 

1 1 

where f* = f^+^/j f ^+'^ dX is the optimal point density for u = fX and 

Va{u\\v) = ^^^log(^y" nV-"dA^ (8) 

denotes the Renyi divergence of order a / 1 between densities u and v. (Thus the loss is always greater 
than one unless fi = u). 

For the entropy-constrained case the main result of |[T5l implies that if {q^) is asymptotically optimal 
for ji = gX, but it is used for v = /A, then 

lim e'-^i('^")z?.(g„) = ^-^e^'^M/)er^i(/ll.). 

Here Vi{f\\g) = V{f\\g) = f / log ^ dX is the Kullback-Leibler divergence (relative entropy) between 
/ and g. From dUl the loss due to mismatch is 

In Theorem [3] we present a result on mismatch for quantization with constrained Renyi entropy of 
order a € (0, 1). The result states that if is asymptotically optimal for jjL = gX, but is applied to 
u = fX, then 

lim e'"S^i-^DM = —^e-^^-^f\\3..) f f_ ^X 

n^oc (l + r)2'^ J Ig^.,) 
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where 
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gP2 

9a,r = X ^^^^ 

with P2 = ^J-a^ (note that go,r = 9*)- The loss due to mismatch can be expressed as 

D (n ) pr{T>l+rifo,r\\ga,r)-Tla{f\\gc,r)) 

The loss can be seen to be always greater than one unless jj, = u (see Remark |6] following Theorem |3]). 
Setting formally a = or a = 1 (or, more precisely, letting a | or a t 1) in the above formula yields 
the known cases Q and Q. 



3 Entropy density and related results 

Throughout this section we assume that /i = gX. For r > 1 and a € [0, 1 + r) \ {1} let 

1 — a + cxr 1 — a -\- r 

h = , ^ , h = , ■ (12) 

1 — a + r 1 — a 

Definition 1. Lef C(r) = ant/ define, for a G [0, 1 + r) \ {1}, 

whenever the integral is finite. Note that Qa,rii^) G (0, 00). We call Qa,rip) the quantization coefficient 
offi. 

Definition 2. A one-dimensional probability density function g is called weakly unimodal if it is contin- 
uous on its support and there exists an Iq > such that {x : g{x) > 1} is a compact interval for every 
I G (0,/o). 

Note that every weakly unimodal density is bounded and its support is a (possibly unbounded) inter- 
val. Clearly, all continuous unimodal densities are weakly unimodal. The class of weakly unimodal den- 
sities includes many parametric source density classes commonly used in modeling information sources 
such as exponential, Laplacian, Gaussian, generalized Gaussian, and all bounded gamma and beta den- 
sities. 

The following is one of the main results in ||2T]| . 

Tlieorem 1 ( II21I Thm 3.4]). For r > 1 and a € (0, 1), if /x has a weakly unimodal density g and 
J d^{x) < 00 for some 6 > 0, then Qa,r{iJ') well defined and 

lim e'^^I)°(i?) = QaAl^). 
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Remark 2. (a) The theorem and Q express the same asymptotic resuh since 



The quantization coefficient Qa,r can also be expressed in terms of Renyi divergences ([8]l and the density 
Qa^r introduced in ([TOl l. One can easily verify that 

/ J {9a,r) 

Furthermore, for any density h with / dX < oo, 

' dX = ilg^rdxT^' li V^\(Mi+0)^A 



/ J \J g^+^ dX, 



5 , 

l + r 

\g\\ 1 e''-^^+''*^^°'''ll''''. 

T+r 



Substituting h = ^(q,^ and combining with ( [T3l ) we obtain 

\ & 



/3l ^\ \ _ ||„|[ , p1-{Vl + r{gQ,r\\9a,r)-'Da{g\\ga,r)) 



l+r 



(b) Theorem 3.4 in 11211 also covers the more exotic a € [— oo, 0) case, but for technical reasons we 
require that a E (0, 1). The weak unimodality condition is a technical one and most likely can be 
significantly relaxed. 

Definition 3. A sequence of quantizer {qn)nen with H^{qn) ^ oo n — > oo called a-asymptotically 
optimal for fi if 

lim = 1. 

Remark 3. In what follows we will simply write "asymptotically optimal" instead of "a-asymptotically 
optimal." Under the conditions of Theorem [H a quantizer sequence (g„) with H°{qn) ^ oo is asymp- 
totically optimal for fi if and only if 

lim e^^M('?")z)^(g„) = Q„^,(^). 
n— >-oo 

For any measurable yl C M with fJ,{A) > we denote by fJ-{-\A) the conditional probability for ^ 
given A. Let c, d € M be such that a < d and /i((c, d]) G (0, 1), but otherwise arbitrary. In the following 
theorem, we let Ai = [a, d], = M \ Ai, and = fJ.{-\Ai) for i G {1, 2}. 
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Theorem 2. Let r > 1 and a G (0, 1). Let /x = g\, where the density function g is weakly unimodal and 
satisfies J dfi{x) < oo for some 5 > 0. Let {qn)n<m be an asymptotically optimal sequence for fi. 

Then, fori G {1,2}, 

^^^^^^^ = t^^^^^^-^- '''' 

and (qn) is asymptotically optimal for /Xj, i.e., lim„_!.oo H'^.{qn) = oo and 

lim e^^-" (^")Z)^,(g„) = QaA^^^)■ (15) 

n— >-oo 

Remark 4. (a) As discussed in Section [231 the ratio on the left hand side of ([Mil can be interpreted as 
the relative contribution to Renyi entropy of interval /. The theorem determines the hmit of this relative 
entropy contribution for a sequence of asymptotically optimal quantizers. The method used in the proof 
is a generalization of the approach developed by Bucklew [7 ] for the case a = 0. 

(b) Using a E (0, 1) and the condition J \x\^~^^ dp,{x) < oo, the integral in the definition of Qa,r{pi') can 
be shown to be finite by an application of Holder's inequality as in W2\ Remark 6.3 (a)]. For the same 
reason, Qa,ripi) is finite for i € {1, 2}. 

In the proof of the theorem we will need the following lemma which is proved in the Appendix. 

Lemma 1. Under the conditions of Theorem\2\the following hold: For i £ {1, 2}, 

lim = oo (16) 

and for all p G M, 

Proof of Theorem |2] We begin the proof by showing that ([T4l) holds if we additionally assume that for 

i G {1,2}, 

limsupe''^^"^^")"^"^^''"" < oo. (18) 
In this case, any subsequence of has a sub-subsequence, which we also denote by (qn), such that 

lim e^(^M(^")-^Mi(5"))=dT^ (19) 

for some d G [0, oo). The obvious bound 

implies that d > 0. In what follows we show that d is independent of the choice of the sub-subsequence 
(and thus the hmit in (fT9l) holds for the original sequence) and exphcitly identify d. 
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For any two sequences and (vn) of positive reals we write u„ ~ v„ if 



lim — = 1. (21) 



Note that if n„ ~ Vn and nj^ ~ v'^, then + nj^) ~ (w„ + u^) and Un ■ u'^ Vn ■ v'^. We can rewrite 
dull as 



d 

We note that 



i=l 



i=l 



Since the cells of g„ are intervals, at most two of them may intersect both Ai = (c, d] and A2 = M \ (c, d] 
(namely, those containing c and d). Then (fTTl) implies 



a£q„{R):q-^{a)cAi 



1-a 



In view of (l22l ) we conclude 



M^2)^ ( 1 - j "^ (24) 



Applying dlU) and (O to (US) we obtain 
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r 

+ KA2f'^' (- ) ' " e'-^M,feO^^^^(g^). (25) 



fi{Ai}-°'d, 



Since H^.{qn) — )• oo by ( fT6l ). Theorem [T]implie; 



liminfe'"^^^«(^")L'^,(g„) >g„,,(/xi), iG{l,2} (26) 



and thus the Umit inferior of the the right hand side of ( [25] ) is lower bounded by 

r 



fj,{Ai)-°'d, 

C(r)(M^i)-"d)^ f / /idAyVc(r)(- ^^V' . (27) 



At(Ai)-«d. 

In view of the definition of Qa,r{l^), combining (l25l) and (l27l) yields 

/32 



I /ma)' 



> F{d) (28) 



where 

r 

F{d) = {^^{A^)-^d)^ ( [ dx) ' + ( ) ' " ( [ /MaV . 

Now let 

r d\ 

do = AAir (29) 
Jai s'^^ dX 

and note that the bound (l20l ) implies G (0, 1]. Moreover, from (fTSl ) we actually obtain 

d~^fi{Ai)°' S (0, 1). Thus if d 7^ do, then Lemma|4]in the Appendix gives F{d) > F{do). Moreover, a 
simple calculation yields F{do) = (J g^^ dX)^^. Hence we deduce from (1281 ) that d = dQ. Because we 
chose an arbitrary convergent subsequence in ([T9l ). we obtain that (fT9l ) actually holds with d = do for the 
original quantizer sequence. This and (l29l ) yield (fT4l) for i = 1. Also, (|24l) and (l29l ) imply ([14)) for i = 2. 

As next step we will prove that ([T5] ) is true under the assumption ([TSl ). We proceed indirectly. Assume 
first that ([T5] ) is not true for i = 1. Then by ([26l ) we can choose a subsequence of ((7„), also denoted by 



'strictly speaking, Theorem[T]( l21l Thm 3.4]) does not apply for fi2 since its density g2 is not weakly unimodal. However, 
g2 is the mixture of two weakly unimodal densities with well-separated supports, and the proof of 1211 Thm 3.4] can easily be 
extended to this case. 
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(qn), such that 



lim e^'^'^i^'^-^D^M > QaAf^i) = C{r) I [ 



I3i \ 

dX 



We deduce from (|23l) and (O that 

nmsupe'^^M2('?")D^^(g„) < C{r)(^l^ (^-j-^Y dxj (30) 



since otherwise we would have 



/32 



which would contradict the asymptotic optimality of But the right hand side of (l30l ) is Qa,r{fJ-2), 
which contradicts (|26l ). so ([T5] ) must hold for i = 1. Similarly, we end in a contradiction if we assume 
that ( [T5] ) does not hold for i = 2. 

It remains to prove that ( fTSl ) must hold. Assuming the contrary, we have 

n— >oo 

Since L';,(gn) > /i(^i)^/^,(9n)> 



lim inf 



which would imply 



liminfe''^^«('?")L>^,(g„) =0 

n— >oo 

contradicting (|26l) . Hence ( fTSl ) must hold and the proof is complete. □ 
Let ((7n)neN be a sequence of quantizers and for any n > 1 and any Borel set C M define 

M^{E) = e'-^MC?") jj:,- q^{x)\' g{x) d\{x). (31) 
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Moreover, for a e [0, 1 + r) \ {1} let 

Mg{E) = C{r) l^j^g^^ dX^ {1^''^^) ' ^^^^ 

Clearly, Mg and Mg are Borel-measures on M that are absolutely continuous with respect to A. We 
define the probability measure fi by setting, for any Borel set E cM., 

KE) = 7 \ (33) 

Corollary 1. Let r > 1 and a € (0, 1). Suppose that fi = gX, where the density function g is weakly 
unimodal and satisfies J diJ,{x) < oo for some 6 > 0. If {qn)n€N cin asymptotically optimal 

sequence of quantizers for fi, then for any c, d € M such that —oo<c<d<oowe have 

/(c,d] l^-9«(2^)r5(2;)dA(x) 
Jr\^ - Qn{x)\'^g{x)dX{x) 

( ii) Mg converges weakly to Mg. 

Remark 5. Combining Theorem |2] and the corollary and using the --^ notation introduced in (|2T]) . we 
observe that 

/mi \x-qn{xWdfl{x) ^ Eagg„(R)/^(gnHa)n(c,d])" ^^^^ 
J^\x-qn{x)\'dfl{x) Eaeg„(R) ^(gn 

This means that the relative error and entropy contributions of over any given interval asymptotically 
coincide. 

Proof of CorollaryU} We start by proving (i). Let A = (c, d] and define 

Ia ~ Qnix)\'''gix) dX{x) 



MA) 



IrIx - qn{x)\''g{x)dX{x)' 



Obviously we can assume without loss of generality that € (0, 1). Applying ([141 ) and ( fT5] ) in 

Theorem 121 we obtain 



e-H^M f^\x-qn{xWgix)dX{x) 
e'^^^'^"^ I \x - qn{x)\'g{x) dX{x) 

Qa,r (/^) 

Qa,r (a*) 

V Ls'^^dA y Q„,,(/i) 



(35) 
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Definition[T] (1331 ). and a straightforward calculation yield that the right hand side of (1351 ) is equal to fi{A). 

Next we prove (ii). Because is asymptotically optimal for fi we have Mg{M.) — > Mg(R.) as 
n — oo. Moreover, Mg is a finite measure. Due to a refined version of the Portmanteau theorem H 
Thm. 2.4 and Example 2.3] it suffices to prove that M^((c, d]) — )■ Mg{{c, d]) for any — oo<c<(i<cxD. 
Let A = (c, d] and assume fji{A) > 0, since otherwise Mg{A) = Mg{A) = for all n. Applying the 
definitions ( [3T] ) and ( [32l ). we obtain 



Mg{A) D^{qn) c{r) (/ gP^ dX) ^ ^4 9'^' dX ' 

Since (qn) is asymptotically optimal for fi and by (i) we deduce 

M-{A) _ C{r){JgP^dXf^^^ 



^132 f^g^^dX 



which proves (ii). □ 

4 Asymptotic mismatch 

In this section we investigate the performance of a sequence of quantizers that is asymptotically 
optimal for the source distribution fj. having density g, but is applied to the source distribution v having 
density /. 

Theorem 3. Let r > 1 and a € (0, 1). Suppose /i = gX, v = fX, where g and f are weakly unimodal 
densities such that f /g is bounded. Assume f \x\^'^^ dfi{x) < oo for some 6 > 0. If {qn)ne'N an 
asymptotically optimal sequence of quantizers for fi, then 

|- _ J(//g)V' dX 

and 

lim e'^^^'^-'^DM 

n— >oo 

= C{r) ( [ r (5^)'-" dx) / dX. (37) 



Remark 6. (a) The mismatch formula (1371 ) is best interpreted through the companding quantization 
approach. In 11211 Remark 4.12] it was shown that for a source with density g, companding quantizers 
having point density h induce high-rate asymptotics performance proportional to 

g"h^-"dX] / gh-^'dX. (38) 
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Asymptotically optimal companding is obtained by setting h = 



-1//32 (J gi/132 dXy^, which is the 



unique minimizer of (|38] ). If the sequence of companding quantizers with this choice of h is now applied 
to the mismatched distribution v = /A, then the same asymptotic performance as in (|37] ) is obtained. 
Thus the main significance of (l37l) is that it holds for an arbitrary asymptotically optimal sequence 
(qn)- The analogy with companding quantization suggests that although can have infinitely many 
codecells, one can interpret ga,r as the point density related to every asymptotically optimal sequence of 
quantizers for /i = gX. 

(b) Using the notation introduced in Sections [2.21 and [241 we can rewrite the mismatch formula ( [37] ) in 
the equivalent forms 



Formula (fTTl) for the loss due to mismatch follows from either of the last two expressions. The loss is 
always greater than one unless ^ = u since, according to the preceding comment, h = fa,r is the unique 
minimizer of e'"(-^i+' (-^o,r||/i)-»Q(/||/i)) oyer all densities h. 

(c) The condition for the boundedness of f /g is the same as in the variable-rate mismatch result of 
lITSl . The fixed-rate result of Bucklew [7] requires essentially the same condition since the only known 
example when the uniform integrability condition given there is satisfied requires that f /gbe bounded. 

(d) The conditions of Theorem [3] are satisfied when the support of fi and u is the same compact interval 
/ and the corresponding densities g and / are continuous and bounded away from zero on /. But the 
theorem may also apply to distributions with unbounded support. For example, if g and / are Gaussian 
or Laplacian densities with mean zero and variance a"^ and a^, respectively, then the conditions are met 
if fj^^j > a^. Unfortunately, the boundedness condition is not satisfied when o"^ < al or when g is 
Gaussian and / is Laplacian. Na ||23l obtained a mismatch result for two zero-mean Laplacian sources 
with arbitrarily mismatched variances by considering quantile quantizers, a special class of fixed-rate 
asymptotically optimal quantizers closely related to companding quantizers. 

Proof. Let / = supp(z^). We will proceed in several steps. 

1 . First we prove relation ( [36l ) under the stated assumptions on g and f /g, but additionally assuming that 
/ is a compact interval and 



Let m > 2 and let {Ik,m ■ k = 1, . . . , m} be a collection of disjoint intervals of equal length \{I)/m 
such that Ua^^i -^fc,m = I- Let Ik^m = inf Ik,m and rk^m = sup Ik,m, denote, respectively, the left and 




l + r 




min{/(x) : X G /} > 0. 



(39) 
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right endpoints of Ik,m- Define 

m 
k=l 

and 

i{f,I) = mm{/(3;) : x G /}, s{f,I) = may:{f{x) : x £ I}. 

Note that caid{Sm,n) < m + l and < i{f, I) < s{f, I) < oo. Since f /g < M for some M < oo, we 
have 

u{A) < Mfx(A) (40) 
for any measurable j4 C M. Thus by ([TT] ) in Lemma[T]we get 

limsup 3| < M limsup ■ — = U. 

Noting that for any a € qnC^) \ Sm,n we either have (?^^(a) C /fc,™. for some k € {1, • • • ,m} or 
i/(g~^(a)) = 0, the above implies 

limsupe(^-")(^-('?")--^M(9n)) 

n— >oo 

= lim sup — j 



y lim sup ^"g-^^W-^^" 



= y limsup VMqn (a)n7fc,„)/^ _ ^^^^ 

fit Eaeg„{R) 

Now we observe that f /g < M and (|39l ) imply for all n > 1, G {1, . . . , m}, and a e (7n(IR), 

S{g,lk,m) ~ IJ.{qn^{a)riIk,m) ~ i{9,h,m) 

Combining (|4TI ) and (l42l ) we deduce from (fT4l) in Theorem |2] that 

limsupe(i-")(^"(^")-^M('^")) < V f fgli^y limsup 



m) 



k 

m 



^ I 4f-h„) V k.„o"' ''^ l-r .. ,,,, 
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where ft is defined in (1331 ) and we liave defined 



77 -V^ 



■s(/,4,m) V 
A:,m) 



Here 1a denotes tlie cliaracteristic function of A C M defined by l^(x) = 1 if x € ^ and 1a{x) = if 
X ^ A. Similarly we obtain 

liminfe^^-^^^^^^^-^-^MC?")) > [ h^dfi (44) 



with 

m 



( i{fJk,m) \°' 



Obviously, < hm, and since /, g, and f / g are continuous on / and the common length of the 
intervals Ik^m converges to zero as m oo, 

lim hm{x) = lim hm{x) = (f{x)/g{x))°' for all x G /. 
Since the hm are uniformly bounded, from Fatou's lemma and by dominated convergence, we get 

/(f/g)°'dfi = / lim inf /i d/i < lim inf / h^dfi 
J m— >oo m-->oo J 

< limsup hmdfi= if/ g)" dfi. (45) 

m— i>c» J J 



Combining ^5} with (143]) and (1441) we obtain 

lim e^i-")^^"^''")-^^^^''")) = lU/gTdfL. 



By the definition of fl in (1331) this yields (1361 ). 

2. We now prove relation (l36l ) under the stated assumptions. Since / is weakly unimodal, the set 
Is = {x : /(x) > (^} C / is a compact interval for all > small enough. Since U5>o ~ 
have u{I \ I^) ^ as 5 — 0, and we also have \ I^) — as 5 — > because fi{-\I) is absolutely 
continuous with respect to v. Consequently, 

lim sup 



I Jls 

< M''{fi{I\Is) + \l-u{Is)-''\Kh)) ^0 (46) 
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as (5 — )• 0. Set [cs, ds] ■= h- Using (l40l ) and ([17] ) in Lemma [T] we obtain 



lim sup 

n—^oo 



lim sup 



< M° lim sup 



/^fe'(9n(c5)))°+/u(g-i(gn(4)))" 



+ M"* lim sup 

n—^oo 



+ lim sup 

n— >oo 



M° lim sup 



+ lim sup 

n— >oo 



< M'^fi{I\Is) 

(1 - K/.)-") Eae.„W: .-(a)C/. K^n ^a) H I^)^ 



+ lim sup 



+ z^(/5)~"M" lim sup 



^(9nn9n(c5)))" + /^(^r7H9n(rf<5)))' 



< M°(/i(/\l5) + l-l/(/5)~")^0 



(47) 



as 5 — > 0. Noting tiiat the density of satisfies tiie condition imposed on / in step 1, we obtain 

from this step that 



n-^oo Jj^ 



(48) 



Combining (|46l).(|47]). and ( |48l ) we obtain that given any e > we can can choose 6 = 6{e) > small 
enough and N = N{6, e) large enough such that for all n > N, 



< £ 



which yields ( [36l ). 

3. We finish the proof by proving assertion (|37] ). Using definition (|3TI ) we get 



if/g)dM^. 
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Thus Corollary [T](ii) and (l36l) yield 



Y.m^e^^^^"-) j\x- ,„(x)r du{x) = {J^Y^T^) / U / 9) dM, 
Using (|32l ) we calculate 

lim e'"^"^^"^ / \x - qn{x)\'' dv{x) 

n— >-co J 



r 

= C{r) Q rg^^-^dX^ {^j fg^^-'dX^ . (49) 

Using definition (fT2)) it is easy to check that /3i — a = and /3i — 1 = — ^> and hence (l49l ) is 

equivalent to (l37l) . completing the proof. □ 



5 Concluding remarks 

We extended point density and mismatch results in fixed and variable-rate asymptotic quantization the- 
ory to scalar quantization with Renyi entropy constraint of order a G (0, 1). We showed that the Renyi 
entropy contribution of a given interval to the overall rate for a sequence of asymptotically optimal 
quantizers is determined by the so-called entropy density of the sequence, an analog of the traditional 
quantizer point density function. A dual of this result quantifies the distortion contribution of a given 
region to the overall distortion. We also proved a mismatch formula for a sequence of asymptotically op- 
timal Renyi entropy constrained scalar quantizers. One can recover the known results for the traditional 
rate definitions by formally setting a = or a = 1 in our density and mismatch results. 

A natural question is whether the density and mismatch results of this paper can be generalized to 
higher dimensional (vector) quantization. To make progress in this direction, one first needs to generalize 
Theorem[T]to higher dimensions (cf. 11211 Section VIII]) to obtain an analog of Zador's fixed and variable- 
rate vector quantization results for Renyi entropy constraint. Assuming one can prove such a result, the 
main difficulty in generalizing our proofs seems to be controlling the entropy contribution at the boundary 
of hypercubes (higher-dimensional intervals). 

Another interesting question is whether the coincidence of distortion and entropy densities described 
by ( [34l ) in Remark |5] is particular to quantization with Renyi entropy or is a deeper phenomenon. In 
particular, one can ask whether replacing Renyi 's entropy with some more general information measure 
(cf. 191) would preserve the existence of and the special relationship between entropy and distortion 
densities. Answers to these questions would provide a more complete understanding of some of the finer 
aspects of quantization theory. 

As mentioned before, an analog of the fixed-rate point density result of Bucklew ||7l (see dUl) can- 



20 



not hold for arbitrary sequences of asymptotically optimal entropy-constrained quantizers. However, 
point densities play an important role in our intuitive understanding of the structure of optimal quan- 
tizers, and may provide (heuristic) guidance in constructing (nearly) optimal quantizers. Thus it would 
be interesting to find a framework within which rigorous point density result can be proved for Renyi 
entropy constrained quantization (and for traditional entropy-constrained quantization). For the scalar 
case, companding quantization provides such a framework, but for higher dimensions, the restriction to 
companding usually precludes asymptotic optimality [5|. 



6 Appendix 

Proof of LemmaUl We first show ([T6l) . The asymptotic optimality of for fj, imphes that D^{qn) 
as n ^ oo. Since /i has a density, this yields, via Lemma |2] below, the intuitively obvious fact that 

lim max{//(g^"'^(a)) : a € = 0. 

ra— )-oo 

This also gives for i € {1,2}, 

lim max{;Uj(g~^(a)) : a G = 0. (50) 

n— ^-oo 

Let p = {pi,P2, . . .) be a probability vector and p^ax = max{pj : i € N}. Since a G (0, 1), we can 
lower bound (p) as 

= -logpmax- 

Combing this bound with (l50l ) yields ([T6l ). 

Next we prove ([TT] ) by contradiction. If the first limit in ([17] ) does not hold, then there is a T > and 
a subsequence of which we also denote by (qn), such that 

lim ^^1nH<ln{p))r ^ ^ ^5^^ 



We have 



Hfliqn) = z log 



1 , fEaeq„iR)f^ilnHa)T^ 



V KQn {qnip))y 
+ log (/^(^n n9n(?')))) 
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where the inequaUty holds since a G (0, 1). Because (g^) is asymptotically optimal, we know that 
^^(.Qn) ^ oo as n — > oo. But the right hand side of (l52l ) converges to a finite limit by assumption (ISTI ). 
a contradiction. 

Also, ( fT6l ) and an argument identical to the proof of the first limit in dTTl) imply that for all |j G M and 

iG{l,2}, 



lim 







which completes the proof. 



□ 



Lemma 2. Assume /x a probability measure on W^, let r > 0, and let \\-\\ be any norm on M'^. Suppose 



(qn) is a sequence of d-dimensional vector quantizers (mappings q„ 
countable) such that 



with qr, 



at most 



lim 

n— >cxD 



/ llx - qn{x)\\'' ^i{dx) = 0. 



Then 



lim max{/i(g„^(a)) : a G gn(M'^)} = 

ra— )-cxD 

if and only if is nonatomic, i.e., ^{{x}) = Ofor all x G M*^. 



(53) 



Proof. If > for some x, then fJ-{qn^ (qnix))) > fi{{x}) shows that (1531 ) cannot hold. Now 

assume that jj, is nonatomic. We proceed indirectly to prove (|53] ). Since f \\x — qn{x)\\^ fj,{dx) > 
/g-i(a) 11^ ~ Qnix)]]"^ fJ-{dx) for all n and a G (7„(M'^), if (|53] ) does not hold, then (considering sub- 
sequences if necessary) there exist an e > 0, points a„ G M'^, and measurable sets An C M"^, such 
that 

lim / \\x — an\\^ ^i{dx) = 0, > e for all n. (54) 

Let B{z, 6) = {x G M.'^ : \\x — z\\ < 6} denote the open ball of radius 6 > centered at z G M*^. We 
have for all 6 > 0, 

/ ||x - anW"" fi{dx) > 6'^iJ,{An \ B{an,6)) 

J An 

which, combined with (l54l ). implies lim„ /i(An \ B{an, 6)) = 0. Thus for all 5 > 0, 

lim'mf fi(B( an, 6)) > e. 

n^oo 

This immediately implies that {a„ : n G N} is a bounded set, since limsup„ ||a„|| = oo would yield 
liminf„ //(i?(an, (5)) = because, as a probability measure on M'^, /i is tight. Thus we can choose a 
subsequence of (a^), which we also denote by (a„), such that a„ ^ a G M'^ as n ^ oo. For this 
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subsequence, B{an, S) C B{a, 26) for all n large enough, implying, for all 6 > 0, 

fi(B{a,26)) > limini fi(B{an,d)) >e. 

n— >-oo 

Since = \\m.s^Q ^{B{a, 2(5)), we obtain /i({a}) > e, which contradicts our assumption that n 

is nonatomic. □ 

Lemma 2>. Let A>Q, B > 0, and define F : (0, 1) ^ M by 

A B 

n^) = ^ + 



Then 

/ 1 1 \ 1+7 

inf{F(z) : z G (0, 1)} = f B~ + A" j 
7/'min(A, i?) > 0, then F{zq) < F{z) for every z G (0, 1) \ {zq}, where 



za = 

Proof. The assertion is obvious for the cases ^ = 0, i? = Oor^ + i? = 0. Thus we can assume that 
^4 > and > 0. But in this case the assertion follows from elementary calculus. □ 

A special case of the following lemma has already been used in Q. For the reader's convenience we 
provide a detailed proof. 

Lemma 4. Let r > 1 and a € (0, 1). Let E cM be measurable. Then, 



gP' dX ] = inf { 



^2 



(1 _ 

///x(F) G (0,1). then 

( fa^^ dxY' < (/^^'^-^^)'^ + {k\E9''dxf 

for every R G (0, 1) \ {Rq}, where Rq = dX/ dX. 

Proof. The assertion follows from Lemma |3] with 

7 = /32 - 1 = r/(l - q) > 0, A = (^j dX^\ wd B = (^j dA^ 
(note that A > and S > if /i(F) G (0, 1)). □ 
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